Skip to content

TEZ-4721: Remove legacy Hadoop 2.x support and hadoop-shim modules#505

Open
Aggarwal-Raghav wants to merge 1 commit into
apache:masterfrom
Aggarwal-Raghav:TEZ-4721
Open

TEZ-4721: Remove legacy Hadoop 2.x support and hadoop-shim modules#505
Aggarwal-Raghav wants to merge 1 commit into
apache:masterfrom
Aggarwal-Raghav:TEZ-4721

Conversation

@Aggarwal-Raghav
Copy link
Copy Markdown
Contributor

@Aggarwal-Raghav Aggarwal-Raghav commented Jun 2, 2026 •
edited
Loading

  • Drop Hadoop 2.x Support by eliminate Hadoop Shim Layer
  • Moved the Hadoop 2.8+ shims logic to TezUtilsInternal.java.
  • Removed hadoop27, hadoop28 profiles
  • Removed GuavaShim layer as well which was pre v18.0 of guava.

This was referenced Jun 2, 2026
Closed
Open
@tez-yetus
Copy link
Copy Markdown

tez-yetus commented Jun 2, 2026

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 10m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 xmllint 0m 1s xmllint was not available.
+0 🆗 shelldocs 0m 1s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 32 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 42s Maven dependency ordering for branch
+1 💚 mvninstall 4m 35s master passed
+1 💚 compile 4m 1s master passed
+1 💚 checkstyle 10m 42s master passed
-1 ❌ mvnsite 0m 43s /branch-mvnsite-tez-api.txt tez-api in master failed.
-1 ❌ mvnsite 0m 40s /branch-mvnsite-tez-runtime-library.txt tez-runtime-library in master failed.
-1 ❌ mvnsite 0m 37s /branch-mvnsite-tez-mapreduce.txt tez-mapreduce in master failed.
-1 ❌ mvnsite 1m 50s /branch-mvnsite-root.txt root in master failed.
-1 ❌ mvnsite 0m 27s /branch-mvnsite-hadoop-shim-impls_hadoop-shim-2.7.txt hadoop-shim-2.7 in master failed.
-1 ❌ javadoc 0m 23s /branch-javadoc-hadoop-shim-impls_hadoop-shim-2.7.txt hadoop-shim-2.7 in master failed.
+0 🆗 spotbugs 0m 41s hadoop-shim in master has 3 extant spotbugs warnings.
+0 🆗 spotbugs 1m 18s tez-api in master has 92 extant spotbugs warnings.
+0 🆗 spotbugs 0m 48s tez-common in master has 13 extant spotbugs warnings.
+0 🆗 spotbugs 0m 54s tez-runtime-internals in master has 92 extant spotbugs warnings.
+0 🆗 spotbugs 1m 10s tez-runtime-library in master has 143 extant spotbugs warnings.
+0 🆗 spotbugs 0m 55s tez-mapreduce in master has 97 extant spotbugs warnings.
+0 🆗 spotbugs 0m 45s tez-examples in master has 2 extant spotbugs warnings.
+0 🆗 spotbugs 1m 39s tez-dag in master has 537 extant spotbugs warnings.
+0 🆗 spotbugs 0m 54s tez-tests in master has 6 extant spotbugs warnings.
+0 🆗 spotbugs 1m 51s tez-plugins in master has 43 extant spotbugs warnings.
+0 🆗 spotbugs 0m 47s tez-plugins/tez-protobuf-history-plugin in master has 7 extant spotbugs warnings.
+0 🆗 spotbugs 0m 42s tez-plugins/tez-yarn-timeline-history in master has 2 extant spotbugs warnings.
+0 🆗 spotbugs 0m 41s tez-plugins/tez-yarn-timeline-history-with-fs in master has 5 extant spotbugs warnings.
+0 🆗 spotbugs 0m 50s tez-tools/analyzers in master has 20 extant spotbugs warnings.
+0 🆗 spotbugs 0m 28s branch/docs no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 6m 26s root in master has 1054 extant spotbugs warnings.
-1 ❌ spotbugs 0m 24s /branch-spotbugs-hadoop-shim-impls_hadoop-shim-2.7.txt hadoop-shim-2.7 in master failed.
+0 🆗 spotbugs 0m 28s branch/tez-dist no spotbugs output file (spotbugsXml.xml)
+0 🆗 spotbugs 0m 45s tez-tools/analyzers/job-analyzer in master has 20 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 13s Maven dependency ordering for patch
+1 💚 mvninstall 3m 43s the patch passed
+1 💚 codespell 1m 34s The patch generated 0 new + 29 unchanged - 1 fixed = 29 total (was 30)
+1 💚 compile 3m 49s the patch passed
+1 💚 javac 3m 49s root generated 0 new + 741 unchanged - 1 fixed = 741 total (was 742)
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 checkstyle 0m 30s The patch passed checkstyle in tez-api
-0 ⚠️ checkstyle 0m 26s /results-checkstyle-tez-common.txt tez-common: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
+1 💚 checkstyle 0m 28s The patch passed checkstyle in tez-runtime-internals
-0 ⚠️ checkstyle 0m 33s /results-checkstyle-tez-runtime-library.txt tez-runtime-library: The patch generated 1 new + 88 unchanged - 0 fixed = 89 total (was 88)
+1 💚 checkstyle 0m 29s The patch passed checkstyle in tez-mapreduce
+1 💚 checkstyle 0m 26s The patch passed checkstyle in tez-examples
+1 💚 checkstyle 0m 44s tez-dag: The patch generated 0 new + 1138 unchanged - 3 fixed = 1138 total (was 1141)
+1 💚 checkstyle 0m 28s The patch passed checkstyle in tez-tests
+1 💚 checkstyle 0m 26s The patch passed checkstyle in tez-ext-service-tests
+1 💚 checkstyle 0m 31s The patch passed checkstyle in tez-plugins
+1 💚 checkstyle 0m 26s The patch passed checkstyle in tez-protobuf-history-plugin
+1 💚 checkstyle 0m 26s The patch passed checkstyle in tez-yarn-timeline-history
+1 💚 checkstyle 0m 27s The patch passed checkstyle in tez-yarn-timeline-history-with-fs
+1 💚 checkstyle 0m 26s The patch passed checkstyle in analyzers
+1 💚 checkstyle 0m 27s The patch passed checkstyle in job-analyzer
+1 💚 checkstyle 0m 23s The patch passed checkstyle in tez-dist
+1 💚 checkstyle 0m 21s The patch passed checkstyle in docs
-0 ⚠️ checkstyle 1m 9s /results-checkstyle-root.txt root: The patch generated 2 new + 1459 unchanged - 17 fixed = 1461 total (was 1476)
-1 ❌ markdownlint 0m 7s /results-markdownlint.txt The patch generated 1 new + 117 unchanged - 2 fixed = 118 total (was 119)
-1 ❌ mvnsite 0m 43s /patch-mvnsite-tez-api.txt tez-api in the patch failed.
-1 ❌ mvnsite 0m 39s /patch-mvnsite-tez-runtime-library.txt tez-runtime-library in the patch failed.
-1 ❌ mvnsite 0m 34s /patch-mvnsite-tez-mapreduce.txt tez-mapreduce in the patch failed.
-1 ❌ mvnsite 1m 36s /patch-mvnsite-root.txt root in the patch failed.
+1 💚 shellcheck 0m 1s No new issues.
+1 💚 javadoc 9m 13s the patch passed
-1 ❌ spotbugs 1m 7s /new-spotbugs-tez-runtime-internals.html tez-runtime-internals generated 7 new + 82 unchanged - 10 fixed = 89 total (was 92)
-1 ❌ spotbugs 1m 54s /new-spotbugs-tez-dag.html tez-dag generated 3 new + 533 unchanged - 4 fixed = 536 total (was 537)
+0 🆗 spotbugs 0m 29s tez-dist has no data from spotbugs
+0 🆗 spotbugs 0m 27s docs has no data from spotbugs
-1 ❌ spotbugs 7m 7s /new-spotbugs-root.html root generated 10 new + 1037 unchanged - 17 fixed = 1047 total (was 1054)
_ Other Tests _
+1 💚 unit 72m 48s root in the patch passed.
+1 💚 asflicense 7m 48s The patch does not generate ASF License warnings.
256m 35s
Reason Tests
SpotBugs module:tez-runtime-internals
Exception thrown in class org.apache.tez.runtime.task.TezChild at new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At TezChild.java:at new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At TezChild.java:[line 189]
new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.credentials At TezChild.java:String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.credentials At TezChild.java:[line 149]
new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.defaultConf At TezChild.java:String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.defaultConf At TezChild.java:[line 141]
new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.objectRegistry At TezChild.java:String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.objectRegistry At TezChild.java:[line 173]
new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.serviceProviderEnvMap At TezChild.java:String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.serviceProviderEnvMap At TezChild.java:[line 145]
new org.apache.tez.runtime.task.TezTaskRunner2(Configuration, UserGroupInformation, String[], TaskSpec, int, Map, Map, Multimap, TaskReporterInterface, ExecutorService, ObjectRegistry, String, ExecutionContext, long, boolean, TezExecutors) may expose internal representation by storing an externally mutable object into TezTaskRunner2.taskReporter At TezTaskRunner2.java:int, Map, Map, Multimap, TaskReporterInterface, ExecutorService, ObjectRegistry, String, ExecutionContext, long, boolean, TezExecutors) may expose internal representation by storing an externally mutable object into TezTaskRunner2.taskReporter At TezTaskRunner2.java:[line 142]
new org.apache.tez.runtime.task.TezTaskRunner2(Configuration, UserGroupInformation, String[], TaskSpec, int, Map, Map, Multimap, TaskReporterInterface, ExecutorService, ObjectRegistry, String, ExecutionContext, long, boolean, TezExecutors) may expose internal representation by storing an externally mutable object into TezTaskRunner2.ugi At TezTaskRunner2.java:int, Map, Map, Multimap, TaskReporterInterface, ExecutorService, ObjectRegistry, String, ExecutionContext, long, boolean, TezExecutors) may expose internal representation by storing an externally mutable object into TezTaskRunner2.ugi At TezTaskRunner2.java:[line 141]
SpotBugs module:tez-dag
new org.apache.tez.dag.app.rm.TaskSchedulerManager(AppContext, DAGClientServer, EventHandler, ContainerSignatureMatcher, WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.appContext At TaskSchedulerManager.java:WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.appContext At TaskSchedulerManager.java:[line 191]
new org.apache.tez.dag.app.rm.TaskSchedulerManager(AppContext, DAGClientServer, EventHandler, ContainerSignatureMatcher, WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.clientService At TaskSchedulerManager.java:WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.clientService At TaskSchedulerManager.java:[line 193]
new org.apache.tez.dag.app.rm.TaskSchedulerManager(AppContext, DAGClientServer, EventHandler, ContainerSignatureMatcher, WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.webUI At TaskSchedulerManager.java:WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.webUI At TaskSchedulerManager.java:[line 195]
SpotBugs module:root
new org.apache.tez.dag.app.rm.TaskSchedulerManager(AppContext, DAGClientServer, EventHandler, ContainerSignatureMatcher, WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.appContext At TaskSchedulerManager.java:WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.appContext At TaskSchedulerManager.java:[line 191]
new org.apache.tez.dag.app.rm.TaskSchedulerManager(AppContext, DAGClientServer, EventHandler, ContainerSignatureMatcher, WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.clientService At TaskSchedulerManager.java:WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.clientService At TaskSchedulerManager.java:[line 193]
new org.apache.tez.dag.app.rm.TaskSchedulerManager(AppContext, DAGClientServer, EventHandler, ContainerSignatureMatcher, WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.webUI At TaskSchedulerManager.java:WebUIService, List, boolean) may expose internal representation by storing an externally mutable object into TaskSchedulerManager.webUI At TaskSchedulerManager.java:[line 195]
Exception thrown in class org.apache.tez.runtime.task.TezChild at new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At TezChild.java:at new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At TezChild.java:[line 189]
new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.credentials At TezChild.java:String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.credentials At TezChild.java:[line 149]
new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.defaultConf At TezChild.java:String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.defaultConf At TezChild.java:[line 141]
new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.objectRegistry At TezChild.java:String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.objectRegistry At TezChild.java:[line 173]
new org.apache.tez.runtime.task.TezChild(Configuration, String, int, String, String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.serviceProviderEnvMap At TezChild.java:String, int, String, String[], Map, ObjectRegistryImpl, String, ExecutionContext, Credentials, long, String, TezTaskUmbilicalProtocol, boolean) may expose internal representation by storing an externally mutable object into TezChild.serviceProviderEnvMap At TezChild.java:[line 145]
new org.apache.tez.runtime.task.TezTaskRunner2(Configuration, UserGroupInformation, String[], TaskSpec, int, Map, Map, Multimap, TaskReporterInterface, ExecutorService, ObjectRegistry, String, ExecutionContext, long, boolean, TezExecutors) may expose internal representation by storing an externally mutable object into TezTaskRunner2.taskReporter At TezTaskRunner2.java:int, Map, Map, Multimap, TaskReporterInterface, ExecutorService, ObjectRegistry, String, ExecutionContext, long, boolean, TezExecutors) may expose internal representation by storing an externally mutable object into TezTaskRunner2.taskReporter At TezTaskRunner2.java:[line 142]
new org.apache.tez.runtime.task.TezTaskRunner2(Configuration, UserGroupInformation, String[], TaskSpec, int, Map, Map, Multimap, TaskReporterInterface, ExecutorService, ObjectRegistry, String, ExecutionContext, long, boolean, TezExecutors) may expose internal representation by storing an externally mutable object into TezTaskRunner2.ugi At TezTaskRunner2.java:int, Map, Map, Multimap, TaskReporterInterface, ExecutorService, ObjectRegistry, String, ExecutionContext, long, boolean, TezExecutors) may expose internal representation by storing an externally mutable object into TezTaskRunner2.ugi At TezTaskRunner2.java:[line 141]
Subsystem Report/Notes
Docker ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-505/1/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense codespell detsecrets markdownlint mvnsite xmllint javac javadoc spotbugs checkstyle shellcheck shelldocs
uname Linux 107462f69a36 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / 330fdc8
Default Java Eclipse Adoptium-21.0.11+10-LTS
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-505/1/testReport/
Max. process+thread count 2125 (vs. ulimit of 5500)
modules C: tez-api tez-common tez-runtime-internals tez-runtime-library tez-mapreduce tez-examples tez-dag tez-tests tez-ext-service-tests tez-plugins tez-plugins/tez-protobuf-history-plugin tez-plugins/tez-yarn-timeline-history tez-plugins/tez-yarn-timeline-history-with-fs tez-tools/analyzers tez-tools/analyzers/job-analyzer tez-dist docs . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-505/1/console
versions git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1 markdownlint=0.46.0 shellcheck=0.11.0
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Aggarwal-Raghav @tez-yetus